Determining a current pose estimate of an aircraft relative to a runway to support the aircraft on approach

ABSTRACT

A method is provided for supporting an aircraft approaching a runway on an airfield. The method includes receiving a sequence of images of the airfield, captured by a camera onboard the aircraft approaching the runway. For at least one image of the sequence of images, the method includes applying the image(s) to a machine learning model trained to predict a pose of the aircraft relative to the runway. The machine learning model is configured to map the image(s) to the pose based on a training set of labeled images with respective ground truth poses of the aircraft relative to the runway. The pose is output as a current pose estimate of the aircraft relative to the runway for use in at least one of monitoring the current pose estimate, generating an alert based on the current pose estimate, or guidance or control of the aircraft on a final approach.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Application No.63/127,526, filed Dec. 18, 2020, entitled Determining a Current PoseEstimate of an Aircraft Relative to a Runway to Support the Aircraft onApproach, the content of which is incorporated herein by reference inits entirety.

TECHNOLOGICAL FIELD

The present disclosure relates generally to aircraft operation and, inparticular, to supporting an aircraft approaching a runway on anairfield.

BACKGROUND

Safe, predictable, and reliable landings are an essential aspect of aneffective aircraft-based transportation system. In many situations,piloted aircraft can only land visually in high visibility and clearweather conditions. While external navigation systems, such asinstrument landing systems (ILS) and global positioning systems (GPS)may be used in some contexts, some situations require visibility andcloud conditions below a specified decision altitude to permit a pilotto land visually. Even in situations where external systems may be used,those external systems are subject to outages, inaccuracies, andinterference, which raise technical challenges to efforts to automatethe landing process.

The technical challenges associated with automating landings orautomatically generating guidance to assist a pilot during a landing arecompounded by multiple factors, such as the variety and variability ofweather and visibility conditions, the availability and feasibilityspecialized equipment at different airfields, and the robustnessrequired of algorithmic approaches to landing operations.

BRIEF SUMMARY

Example implementations of the present disclosure are directed toaircraft operation and, in particular, to supporting an aircraftapproaching a runway on an airfield. Example implementations address andovercome technical challenges associated with autonomous approach andlanding operations in aircraft using either or both computer visionheuristics or machine learning to determine estimate the pose of theaircraft relative to the runway. Some example implementations areflexible in at least the sense that such example implementations mayaccommodate a range of passive and active sensors, including but notlimited to visual imaging devices, hyperspectral imaging devices, LIDAR,RADAR, and the like, for example, to perceive the landing environmentand to determine the aircraft-relative pose, and may be used with avariety of manned and un-manned aircraft of varying size to land onvisual, precision, and non-precision runways. Some exampleimplementations are also cost-effective, at least in the sense that suchexample implementations do not require expensive in-groundinfrastructure to be installed at a given airfield.

The present disclosure thus includes, without limitation, the followingexample implementations.

Some example implementations provide a method of supporting an aircraftapproaching a runway on an airfield, the method comprising: receiving asequence of images of the airfield, captured by a camera onboard theaircraft approaching the runway; and for at least one image of thesequence of images, applying the at least one image to a machinelearning model trained to predict a pose of the aircraft relative to therunway, the machine learning model configured to map the at least oneimage to the pose based on a training set of labeled images withrespective ground truth poses of the aircraft relative to the runway;and outputting the pose as a current pose estimate of the aircraftrelative to the runway for use in at least one of monitoring the currentpose estimate, generating an alert based on the current pose estimate,or guidance or control of the aircraft on a final approach.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, applying the at least one image to the machine learningmodel includes: applying the at least one image to the machine learningmodel trained to predict a pose of the at least one camera in cameracoordinates; and transforming the camera coordinates for the at leastone camera to corresponding runway-framed local coordinates and therebypredict the pose of the aircraft relative to the runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the at least one image and the labeled images are in anon-visible light spectrum.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the labeled images are mono-channel images, the atleast one image is a multi-channel image, and the method furthercomprises converting the multi-channel image to a mono-channel imagethat is applied to the machine learning model.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the method further comprises cropping the at least oneimage to reduce a field of view of the at least one camera, andmagnifying only a portion of the at least one image on which the runwayis located, before the at least one image is applied to the machinelearning model.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the method further comprises generating the trainingset of labeled images, including at least: receiving earlier images ofthe airfield, captured by the at least one camera onboard the aircraftor a second aircraft approaching the runway, and the respective groundtruth poses of the aircraft or the second aircraft relative to therunway; and labeling the earlier images with the respective ground truthposes of the aircraft to generate the training set of labeled images.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the method further comprises generating the trainingset of labeled images, including at least: executing a flight simulatorconfigured to artificially re-create flight of the aircraft approachingthe runway on the airfield; capturing synthetic images of the airfield,and determining the respective ground truth poses of the aircraftrelative to the runway, from the flight simulator; and labeling thesynthetic images with the respective ground truth poses of the aircraftto generate the training set of labeled images.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, applying the at least one image to the machine learningmodel includes applying the at least one image to machine learningmodels trained to predict respective components of the pose of theaircraft relative to the runway, the machine learning models configuredto determine values of the components and thereby the pose of theaircraft relative to the runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, applying the at least one image to the machine learningmodel includes applying the at least one image to machine learningmodels trained to predict multiple current pose estimates according todifferent algorithms, and the method further comprises: determiningconfidence intervals associated with respective ones of the multiplecurrent pose estimates; and performing a sensor fusion of the multiplecurrent pose estimates using the confidence intervals to determine thecurrent pose estimate of the aircraft relative to the runway.

Some example implementations provide a method of supporting an aircraftapproaching a runway on an airfield, the method comprising: receiving asequence of images of the airfield, captured by at least one cameraonboard the aircraft approaching the runway; and for at least an imageof the sequence of images, performing an object detection andsegmentation in which at least one of the runway on the airfield, or arunway marking on the runway, is detected in the image, and in which amask is produced that includes a segment of pixels of the image assignedto an object class for the runway or the runway marking; determining acurrent pose estimate of the aircraft relative to the runway or therunway marking based on the mask; and outputting the current poseestimate for use in at least one of monitoring the current poseestimate, generating an alert based on the current pose estimate, orguidance or control of the aircraft on a final approach.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, determining the current pose estimate of the aircraftrelative to the runway or the runway marking includes at least:determining the current pose estimate of the at least one camera incamera coordinates; and transforming the camera coordinates for the atleast one camera to corresponding runway-framed local coordinates thatare output for use in the at least one of monitoring the current poseestimate, generating the alert based on the current pose estimate, orguidance or control of the aircraft.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the sequence of images are in the non-visible lightspectrum.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the image is a multi-channel image, and the methodfurther comprises converting the multi-channel image to a mono-channelimage on which the object detection and segmentation is performed.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the method further comprises cropping the image toreduce a field of view of the at least one camera, and magnifying only aportion of the image on which the runway or the runway marking islocated, before the object detection and segmentation is performed.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the method further comprises: accessing an earlierimage in the sequence of images, and an earlier mask includes arespective segment of pixels of the earlier image assigned to the objectclass for the runway or the runway marking; and identifying a portion ofthe earlier image that frames the earlier mask, and thereby the portionof the airfield on which the runway or the runway marking is located,and wherein cropping the image includes cropping the image to a portionof the image that corresponds to the portion of the earlier image.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, performing the object detection and segmentationincludes applying the image to a machine learning model trained toperform the object detection and segmentation.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, performing the object detection and segmentationincludes performing a feature detection in which features of the runwayor the runway marking are detected in the image, and in which the maskis produced from the features.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, determining the current pose estimate includes atleast: applying the mask to a corner detector to detect interest pointson the mask and thereby the runway or the runway marking in the image;matching the interest points on the runway or the runway marking in theimage, to corresponding points on the runway or the runway marking thathave known runway-framed local coordinates; and performing aperspective-n-point (PnP) estimation, using the interest points and theknown runway-framed local coordinates, to determine the current poseestimate of the at least one camera and thereby the aircraft relative tothe runway or the runway marking.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, performing the object detection and segmentationincludes applying the image to a machine learning model trained toperform the object detection and segmentation, and the method furthercomprises and the method further comprises: determining an operationalstate of the aircraft; and based on the operational state, selecting afirst machine learning model or a second machine learning model as themachine learning model to which the image is applied, the first machinelearning model trained to perform a one-class object detection andsegmentation, and the second machine learning model trained to perform amulti-class object detection and segmentation.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, performing the object detection and segmentationincludes applying the image to a machine learning model trained toperform the object detection and segmentation, and the machine learningmodel further determines a confidence interval or multi-dimensionalconfidence matrix associated with detection of the runway or the runwaymarking, and wherein the method further comprises applying the mask to aconfidence filter that passes the mask to the corner detector only whenthe confidence interval is above a threshold confidence interval that isdynamically set based on an operational state of the aircraft.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the PnP estimation is performed to determine tomultiple current pose estimates of the aircraft relative to the runwayor the runway marking, and the method further comprises: determiningconfidence intervals associated with respective ones of the multiplecurrent pose estimates; and performing a sensor fusion of the multiplecurrent pose estimates using the confidence intervals to determine thecurrent pose estimate of the aircraft relative to the runway or therunway marking.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the object detection and segmentation is performed todetect the runway, and in which the mask includes the segment of pixelsof the image assigned to the object class for the runway, anddetermining the current pose estimate includes at least: applying themask to a corner detector to detect a pair of interest points on themask and thereby the runway in the image; matching the pair of interestpoints on the runway in the image, to corresponding points on the runwaythat have known runway-framed local coordinates; and performing aperspective-n-point (PnP) estimation for n=2, using the pair of interestpoints and the known runway-framed local coordinates, to determine thecurrent pose estimate of the at least one camera and thereby theaircraft relative to the runway, the PnP estimation modified based on anassumption that the runway is rectangular and planar, and wherein thePnP estimation is performed as modified to determine the current poseestimate including two degrees-of-freedom (DOF) made up of a verticalangular deviation of the aircraft from a glideslope, and a lateralangular deviation of the aircraft from a centerline of the runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the object detection and segmentation is performed todetect the runway, and in which the mask includes the segment of pixelsof the image assigned to the object class for the runway, anddetermining the current pose estimate includes at least: performing aparameterization of the mask in which a shape of the mask is describedby values of at least one of a set of parameters or an interpolation ofthe set of parameters; and determining a current pose estimate of theaircraft relative to the runway based on the values of the set ofparameters and an expression that maps the at least one of the set ofparameters or the interpolation of the set of parameters to a pose or aninterpolated pose of the aircraft relative to the runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the set of parameters includes moment invariants of themask, and performing the parameterization includes determining values ofthe moment invariants of the mask.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the current pose estimate is determined based on theexpression that is implemented as a lookup table of different values ofthe set of parameters and ground truth poses of the aircraft relative tothe runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the current pose estimate is determined based on theexpression that is implemented as a stochastic algorithm.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the current pose estimate is determined based on theexpression that is implemented as a deterministic algorithm.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the object detection and segmentation is performed todetect the runway, and in which the mask includes the segment of pixelsof the image assigned to the object class for the runway, anddetermining the current pose estimate includes at least: determiningedges of the mask that correspond to sides of the runway, and values ofspatial dimensions of the edges; and performing a regression analysis inwhich a pose of the aircraft relative to the runway is expressed as afunction of parameters that describe spatial dimensions of the edges,the regression analysis performed to determine the current pose estimateof the aircraft relative to the runway from the values of the spatialdimensions of the edges.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, performing the regression analysis includes performingthe regression analysis in which components of the pose of the aircraftare expressed as respective functions of parameters that describe thespatial dimensions of the edges, the regression analysis performed todetermine values of the components and thereby the current pose estimateof the aircraft relative to the runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, performing the regression analysis includes: performingthe regression analysis of different statistical models to determinemultiple current pose estimates of the aircraft relative to the runwayfrom the values of the spatial dimensions of the edges; determiningconfidence intervals associated with respective ones of the multiplecurrent pose estimates; and performing a sensor fusion of the multiplecurrent pose estimates using the confidence intervals to determine thecurrent pose estimate of the aircraft relative to the runway.

In some example implementations of the method of any preceding exampleimplementation, or any combination of any preceding exampleimplementations, the object detection and segmentation is performed todetect the runway, and in which the mask includes the segment of pixelsof the image assigned to the object class for the runway, anddetermining the current pose estimate includes at least: determiningedges of the mask that correspond to sides of the runway, and angulardeviations of the edges from a centerline of the image; and determiningthe current pose estimate of the aircraft relative to the runway fromthe edges and the angular deviations of the edges from the centerline,the current pose estimate including two degrees-of-freedom (DOF) made upof a vertical angular deviation of the aircraft from a glideslope, and alateral angular deviation of the aircraft from a centerline of therunway.

Some example implementations provide an apparatus for supporting anaircraft approaching a runway on an airfield, the apparatus comprising amemory configured to store computer-readable program code; andprocessing circuitry configured to access the memory, and execute thecomputer-readable program code to cause the apparatus to at leastperform the method of any preceding example implementation, or anycombination of any preceding example implementations.

Some example implementations provide a computer-readable storage mediumfor supporting an aircraft approaching a runway on an airfield, thecomputer-readable storage medium being non-transitory and havingcomputer-readable program code stored therein that, in response toexecution by processing circuitry, causes an apparatus to at leastperform the method of any preceding example implementation, or anycombination of any preceding example implementations.

These and other features, aspects, and advantages of the presentdisclosure will be apparent from a reading of the following detaileddescription together with the accompanying figures, which are brieflydescribed below. The present disclosure includes any combination of two,three, four or more features or elements set forth in this disclosure,regardless of whether such features or elements are expressly combinedor otherwise recited in a specific example implementation describedherein. This disclosure is intended to be read holistically such thatany separable features or elements of the disclosure, in any of itsaspects and example implementations, should be viewed as combinableunless the context of the disclosure clearly dictates otherwise.

It will therefore be appreciated that this Brief Summary is providedmerely for purposes of summarizing some example implementations so as toprovide a basic understanding of some aspects of the disclosure.Accordingly, it will be appreciated that the above described exampleimplementations are merely examples and should not be construed tonarrow the scope or spirit of the disclosure in any way. Other exampleimplementations, aspects and advantages will become apparent from thefollowing detailed description taken in conjunction with theaccompanying figures which illustrate, by way of example, the principlesof some described example implementations.

BRIEF DESCRIPTION OF THE FIGURE(S)

Having thus described example implementations of the disclosure ingeneral terms, reference will now be made to the accompanying figures,which are not necessarily drawn to scale, and wherein:

FIG. 1 illustrates an aircraft according to some example implementationsof the present disclosure;

FIGS. 2A and 2B illustrate an aircraft approaching a runway on anairfield, according to some example implementations;

FIGS. 3 and 4 illustrate systems for supporting an aircraft approachinga runway on an airfield, according to various example implementations;

FIGS. 5A and 5B an image of a runway, and a mask of the runway that maybe produced from an object detection and segmentation of the image,according to some example implementations;

FIGS. 6A, 6B and 7 illustrate masks of a runway according to variousapproaches of example implementations of the present disclosure;

FIGS. 8A, 8B, 8C, 8D, 8E and 8F are flowcharts illustrating varioussteps in a method of supporting an aircraft approaching a runway on anairfield, according to some example implementations;

FIGS. 9A, 9B, 9C, 9D, 9E, 9F, 9G, 9H, 9I, 9J, 9K, 9L, 9M and 9N areflowcharts illustrating various steps in a method of supporting anaircraft approaching a runway on an airfield, according to other exampleimplementations; and

FIG. 10 illustrates an apparatus according to some exampleimplementations.

DETAILED DESCRIPTION

Some implementations of the present disclosure will now be describedmore fully hereinafter with reference to the accompanying figures, inwhich some, but not all implementations of the disclosure are shown.Indeed, various implementations of the disclosure may be embodied inmany different forms and should not be construed as limited to theimplementations set forth herein; rather, these example implementationsare provided so that this disclosure will be thorough and complete, andwill fully convey the scope of the disclosure to those skilled in theart. Like reference numerals refer to like elements throughout.

Unless specified otherwise or clear from context, references to first,second or the like should not be construed to imply a particular order.A feature described as being above another feature (unless specifiedotherwise or clear from context) may instead be below, and vice versa;and similarly, features described as being to the left of anotherfeature else may instead be to the right, and vice versa. Also, whilereference may be made herein to quantitative measures, values, geometricrelationships or the like, unless otherwise stated, any one or more ifnot all of these may be absolute or approximate to account foracceptable variations that may occur, such as those due to engineeringtolerances or the like.

As used herein, unless specified otherwise or clear from context, the“or” of a set of operands is the “inclusive or” and thereby true if andonly if one or more of the operands is true, as opposed to the“exclusive or” which is false when all of the operands are true. Thus,for example, “[A] or [B]” is true if [A] is true, or if [B] is true, orif both [A] and [B] are true. Further, the articles “a” and “an” mean“one or more,” unless specified otherwise or clear from context to bedirected to a singular form. Furthermore, it should be understood thatunless otherwise specified, the terms “data,” “content,” “digitalcontent,” “information,” and similar terms may be at times usedinterchangeably.

Example implementations of the present disclosure are directed to theautonomous detection of runways (including heliports) and runwaymarkings. To address and overcome technical challenges associated withautonomous approach and landing operations in aircraft, exampleimplementations of the present disclosure provide a number of visualpose estimation approaches that rely on either or both computer visionheuristics or machine learning. Example implementations determine acurrent pose estimate of the aircraft relative to a runway for use in atleast one of monitoring the current pose estimate, generating an alertbased on the current pose estimate, or guidance or control of theaircraft on a final approach.

Example implementations of the present disclosure are directed toaircraft operation. FIG. 1 illustrates one type of aircraft 100 that maybenefit from example implementations of the present disclosure. Asshown, the aircraft includes an airframe 102 with a fuselage 104, wings106 and tail 108. The aircraft also includes a plurality of high-levelsystems 110 such as a propulsion system. In the particular example shownin FIG. 1, the propulsion system includes two wing-mounted engines 112.In other embodiments, the propulsion system can include otherarrangements, for example, engines carried by other portions of theaircraft including the fuselage and/or the tail. The high-level systemsmay also include an electrical system 114, hydraulic system 116 and/orenvironmental system 118. Any number of other systems may be included.

FIGS. 2A and 2B illustrate an aircraft 200 such as aircraft 100approaching a runway 202 on an airfield 204, according to some exampleimplementations of the present disclosure. As also shown, the runwayincludes runway markings 206 such as threshold markings (piano keys),runway designation markings, touchdown zone markings, aiming pointmarkings (aim pads), runway edge lines, centerline markings, incrementdistance markings, and the like.

As the aircraft 200 approaches the runway 202—and particularly on afinal approach—it may be useful to estimate the position and orientationof the aircraft relative to the runway. This position and orientation isreferred to as the pose of the aircraft relative to the runway, and itmay be used for monitoring, generating an alert, and/or guidance orcontrol of the aircraft.

The pose of the aircraft 200 relative to the runway may be expressed ina number of different manners, and with a number of different componentsthat correspond to degrees of freedom (DOF). In various examples, thepose of the aircraft may be expressed in six degrees of freedom (6DOF)by its position in coordinates on three principal axes including alongitudinal axis (roll axis, x-axis), vertical axis (yaw axis, y-axis)and transverse axis (pitch axis, z-axis), as well as its orientationexpressed as rotation (yaw, pitch, roll) around the principal axes.

In some examples, the pose of the aircraft 200 may be expressed in twodegrees of freedom (2DOF) relative to a glideslope 208 and a centerline210 of the runway 202. The glideslope is set as a predefined angle abovehorizontal (ground level), such as 3° above horizontal, and it may alsobe referred to as the glide path angle (GPA). The pose of the aircraftin some of these examples may be expressed as a vertical angulardeviation OVERT of the aircraft from the glideslope, and a lateralangular deviation SLAT of the aircraft from the centerline of therunway. A vector 216 extends from the aircraft to a glide path interceptpoint 218; and the vertical angular deviation is the difference betweenan angle between the vector and horizontal, and the glide path angle.Similarly, a vector 220 extends from the aircraft to an azimuthreference point 222; and the lateral angular deviation is the anglebetween the runway center point and the azimuth reference point.

In yet other examples, the pose of the aircraft 200 may be expressedsome combination of the components of either or both of theabove-described 6DOF and 2DOF. In particular, for example, the pose ofthe aircraft relative to the runway 202 may be expressed in 3DOF as acombination of an x coordinate (distance to the runway), y coordinate(altitude), and lateral angular deviation θ_(LAT).

FIGS. 3 and 4 illustrate systems 300, 400 for supporting an aircraft 200approaching a runway 202 on an airfield 204, according to exampleimplementations of the present disclosure. The systems may include anyof a number of different subsystems (each an individual system) forperforming one or more functions or operations. One or more of thesubsystems may be located onboard the aircraft, or remote from theaircraft such as at an operations center of the aircraft. The subsystemsmay be co-located or directly coupled to one another, or in someexamples, various ones of the subsystems may communicate with oneanother across one or more computer networks. Similarly, the aircraftmay communicate with one or more of the subsystems across the one ormore computer networks, which during flight may be facilitated by one ormore artificial satellites, ground stations and the like.

Although shown as part of the systems 300, 400, it should be understoodthat any one or more of the subsystems may function or operate as aseparate system without regard to any of the other subsystems. It shouldalso be understood that the systems may include one or more additionalor alternative subsystems than those shown in the figures.

The systems 300, 400 of example implementations are generally configuredto determine pose estimates of the aircraft 200 relative to the runway202. The systems may operate similar to a human pilot on runway approachor takeoff, and may be deployed in autonomous, semi-autonomous oropen-loop operation. The systems may operate independent of radionavigation, such as ILS, GPS and the like. The pose estimates may beprovided in any of a number of degrees of freedom, which may in someexamples be configurable. The system in some implementations maydetermine a pose estimate even when specific runway markings or otherfeatures are not visible in an image, which may enable the system towork for both takeoff and landing.

More particularly, the system 300 shown in FIG. 3 is generallyconfigured to use images and machine learning to determine poseestimates of the aircraft 200 relative to the runway 202. As shown, thesystem of some example implementations includes a pose-estimation engine302, an image processor 304 and a machine learning (ML) model generator306. The pose-estimation engine is configured to receive a sequence ofimages 308 of the airfield, captured by at least one camera 310 onboardthe aircraft 200 approaching the runway 202. The camera may be any of anumber of different types of camera capable of capturing a sequence ofimages, including but not limited to visual imaging devices,hyperspectral imaging devices, LIDAR imaging devices, RADAR imagingdevice, and the like. In various examples, the camera is located onboardthe aircraft in a configuration that allows the camera to capture a viewof the environment ahead of the aircraft in the direction of travel ofthe aircraft.

For at least one image of the sequence of images, the pose-estimationengine 302 is configured to apply the at least one image to a machinelearning model 312 trained to predict a pose of the aircraft 200relative to the runway 202. In this regard, the machine learning modelis configured to map the at least one image to the pose based on atraining set 314 of labeled images with respective ground truth poses ofthe aircraft relative to the runway. For a pose expressed in 6DOF, thepose and the respective ground truth poses may be expressed in x, y, zcoordinates, and yaw, pitch, roll angles; and for 2DOF, the pose and therespective ground truth poses may be expressed in vertical and lateralangular deviation.

The pose-estimation engine 302 is configured to output the pose as acurrent pose estimate 316 of the aircraft 200 relative to the runway 202for use in at least one of monitoring the current pose estimate,generating an alert based on the current pose estimate, or guidance orcontrol of the aircraft on a final approach.

In some examples, the machine learning model 312 is expressed as acustom deep neural network such as a convolutional neural network (CNN).The machine learning model may therefore be designed to include a fullimage-resolution input layer (e.g., 1024×768 pixels), and allow forconfigurable depth to enable optimization based on implementation. Themachine learning model may include a CNN, as well as a fully-connecteddense network and an n-DOF regression, such as a 6DOF regression or a2DOF regression.

In some examples, the pose-estimation engine 302 is configured to applythe at least one image of the sequence of images 308 to the machinelearning model trained to predict a pose of the at least one camera incamera coordinates. In some of these examples, the pose-estimationengine is configured to transform the camera coordinates for the atleast one camera to corresponding runway-framed local coordinates andthereby predict the pose of the aircraft relative to the runway 202.This transformation may be known prior to deployment of the system 300,and correct for any aircraft offset.

In some examples, the of the sequence of images 308 and the labeledimages of the training set 314 are in a non-visible light spectrum. Inother examples, the labeled images are mono-channel images, and the atleast one image is a multi-channel image. In some of these examples, theimage processor 304 is configured to convert the multi-channel image toa mono-channel image that is applied to the machine learning model 312.

In some examples, the image processor 304 is configured to crop the atleast one image of the sequence of images 308 to reduce a field of viewof the at least one camera 310, and magnify only a portion of the atleast one image on which the runway 202 is located, before the at leastone image is applied to the machine learning model 312.

In some examples, the ML model generator 306 is configured to generatethe training set 314 of labeled images. In some of these examples, theML model generator is configured to receive earlier images 318 of theairfield, captured by the at least one camera 310 onboard the aircraftor a second aircraft approaching the runway 202. The ML model generatoris also configured to receive the respective ground truth poses 320 ofthe aircraft or the second aircraft relative to the runway, which may bedetermined from radio navigation. The ML model generator, then, may beconfigured to label the earlier images with the respective ground truthposes of the aircraft to generate the training set of labeled images.

In some examples, the ML model generator 306 configured to generate thetraining set 314 of labeled images includes the ML model generatorconfigured to execute a flight simulator 322 configured to artificiallyre-create flight of the aircraft approaching the runway 202 on theairfield. In some of these examples, the ML model generator isconfigured to capture synthetic images 324 of the airfield, anddetermine the respective ground truth poses 320 of the aircraft relativeto the runway, from the flight simulator. And in some of these examples,the ML model generator is configured to label the synthetic images withthe respective ground truth poses of the aircraft to generate thetraining set of labeled images.

In some examples, the pose-estimation engine 302 is configured to applythe at least one image of the sequence of images 308 to machine learningmodels 312 trained to predict respective components of the pose of theaircraft relative to the runway 202. In some of these examples, themachine learning models are configured to determine values of thecomponents and thereby the pose of the aircraft relative to the runway,such as in a number of DOF.

In some examples, the pose-estimation engine 302 is configured to applythe at least one image of the sequence of images 308 to machine learningmodels 312 trained to predict multiple current pose estimates 316according to different algorithms. In some of these examples, thepose-estimation engine is configured to determine confidence intervalsassociated with respective ones of the multiple current pose estimates.The pose-estimation engine is then configured to perform a sensor fusionof the multiple current pose estimates using the confidence intervals todetermine the current pose estimate of the aircraft relative to therunway 202.

Turning to FIG. 4, similar to system 300, the system 400 is alsogenerally configured to use images to determine pose estimates of theaircraft 200 relative to the runway 202. The system 400 is configured toperform an object detection and segmentation of an image to detect therunway and/or a runway marking, and produce a mask of pixels of theimage assigned to an object class for the runway or runway marking. Thesystem is then configured to determine a current pose estimate of theaircraft based on the mask. In various examples, the system may performeither or both the object detection and segmentation, or determine thecurrent pose estimate, according to a variety of techniques.

Similar to the system 300 of FIG. 3, the system 400 of FIG. 4 includes apose-estimation engine 402 and an image processor 404. Thepose-estimation engine is configured to receive a sequence of images 406of the airfield, captured by at least one camera 408 onboard theaircraft 200 approaching the runway 202. The camera 408 may be the sameas or similar to the camera 310. FIG. 5A is one example of a suitableimage 500 of a runway 502 including runway markings 504. For at leastone image of the sequence of images, the pose-estimation engine isconfigured to perform an object detection and segmentation in which atleast one of the runway on the airfield, or a runway marking on therunway, is detected in the image. This may include a pixel-wise instancesegmentation.

As shown, mask 410 is also produced from the object detection andsegmentation. The mask includes a segment of pixels of the imageassigned to an object class for the runway 202 or the runway marking222. FIG. 5B illustrates a mask 506 of the runway 502 that may beproduced from an object detection and segmentation of the image 500shown in FIG. 5A. The pose-estimation engine is configured to determinea current pose estimate 412 of the aircraft relative to the runway orthe runway marking based on the mask, which may be independent of radionavigation. And pose-estimation engine is configured to output thecurrent pose estimate for use in at least one of monitoring the currentpose estimate, generating an alert based on the current pose estimate,or guidance or control of the aircraft on a final approach.

In some examples, the pose-estimation engine 402 is configured todetermine the current pose estimate of the at least one camera in cameracoordinates. In some of these examples, the pose-estimation engine isconfigured to transform the camera coordinates for the at least onecamera to corresponding runway-framed local coordinates that are outputfor use in the at least one of monitoring the current pose estimate,generating the alert based on the current pose estimate, or guidance orcontrol of the aircraft.

In some examples, the of the sequence of images 406 are in a non-visiblelight spectrum. In other examples, the image is a multi-channel image.In some of these examples, the image processor 404 is configured toconvert the multi-channel image to a mono-channel image on which theobject detection and segmentation is performed.

In some examples, the image processor 404 is configured to crop the atleast one image of the sequence of images 406 to reduce a field of viewof the at least one camera 408, and magnify only a portion of the atleast one image on which the runway 202 is located, before the objectdetection and segmentation is performed.

In some further examples, the image processor 404 is configured toaccess an earlier image in the sequence of images 406, and an earliermask includes a respective segment of pixels of the earlier imageassigned to the object class for the runway 202 or the runway marking222. In some of these examples, the image processor is configured toidentify a portion of the earlier image that frames the earlier mask,and thereby the portion of the airfield on which the runway 202 or therunway marking is located. And the image processor is configured to cropthe image to a portion of the image that corresponds to the portion ofthe earlier image.

In some examples, the pose-estimation engine 402 configured to performthe object detection and segmentation includes the pose-estimationengine configured to apply the image to a machine learning model 414trained to perform the object detection and segmentation. In someexamples, the pose-estimation engine configured to perform the objectdetection and segmentation includes the pose-estimation engineconfigured to perform a feature detection 416 in which features of therunway 202 or the runway marking 222 are detected in the image, and inwhich the mask 410 is produced from the features.

Perception-Based Approach. In some examples, the pose-estimation engine402 configured to determine the current pose estimate 412 includes thepose-estimation engine configured to apply the mask 410 to a cornerdetector 418 to detect interest points on the mask and thereby therunway 202 or the runway marking 222 in the image. Examples of suitableinterest points include points on the mask that correspond to points onthe runway such as corners of the runway, points on one or more of therunway markings (e.g., center points), and the like. In some of theseexamples, the pose-estimation engine is configured to match the interestpoints on the runway or the runway marking in the image, tocorresponding points on the runway or the runway marking that have knownrunway-framed local coordinates. And the pose-estimation engine isconfigured to perform a perspective-n-point (PnP) estimation 420, usingthe interest points and the known runway-framed local coordinates, todetermine the current pose estimate of the at least one camera andthereby the aircraft 200 relative to the runway or the runway marking.PnP is a well-established computer vision algorithm that may determine apose estimate given a mapping of pixel coordinates to knownrunway-framed local coordinates. Some implementations of PnP require aminimum of two to four points, while others require more points.

In some further examples in which the pose-estimation engine 402configured to apply the image to the machine learning model 414, thepose-estimation engine is configured to determine an operational stateof the aircraft. Based on the operational state, the pose-estimationengine is configured to select a first machine learning model 414A or asecond machine learning model 414B as the machine learning model towhich the image is applied. In some of these examples, the first machinelearning model is trained to perform a one-class object detection andsegmentation, and the second machine learning model trained to perform amulti-class object detection and segmentation.

In some examples in which the image is applied to the machine learningmodel 414 trained to perform the object detection and segmentation, themachine learning model further determines a confidence interval ormulti-dimensional confidence matrix associated with detection of therunway 202 or the runway marking 222. In some of these examples, thepose-estimation engine 402 is configured to apply the mask 410 to aconfidence filter 422 that passes the mask to the corner detector 418only when the confidence interval is above a threshold confidenceinterval that is dynamically set based on an operational state of theaircraft 200.

The PnP estimation 420 of some examples is performed to determine tomultiple current pose estimates 412 of the aircraft 200 relative to therunway 202 or the runway marking 222. In some of these examples, thepose-estimation engine 402 is configured to determine confidenceintervals associated with respective ones of the multiple current poseestimates. The pose-estimation engine 402 is configured to perform asensor fusion of the multiple current pose estimates using theconfidence intervals to determine the current pose estimate of theaircraft relative to the runway or the runway marking.

Modified PnP Approach. The PnP estimation 420 may be used to determinethe current pose estimate 412 from n corresponding points. Given a knowncalibration of the camera 408 and the location of four projected points,quadratic equations for pose estimation may be postulated. The quadraticequations may lead to 2⁴ solutions for their parameters (with fourpossible solutions with positive z-values). In some PnP algorithms, themost likely set of parameters may be determined with iterative orgeometric methods. According to a modified PnP approach, a-prioriknowledge that applies to all runways can be leveraged to reduce thenumber of points needed, reduce computational cost and solutionambiguity. This approach is premised on an assumption that the runway202 is rectangular and planar.

According to the modified PnP approach of some examples in which theobject detection and segmentation is performed to detect the runway 202,the pose-estimation engine 402 is configured to apply the mask 410 tothe corner detector to detect a pair of interest points on the mask andthereby the runway in the image. The pose-estimation engine isconfigured to match the pair of interest points on the runway in theimage, to corresponding points on the runway that have knownrunway-framed local coordinates. The pose-estimation engine isconfigured to perform the PnP estimation for n=2, using the pair ofinterest points and the known runway-framed local coordinates, todetermine the current pose estimate of the at least one camera andthereby the aircraft 200 relative to the runway.

As indicated above, the PnP estimation 420 is modified based on anassumption that the runway 202 is rectangular and planar. The assumptionthat the runway is planar may simplify y coordinates of corners of therunway (interest points) to all y=0. The assumption that the runway isrectangular may simplify x, z coordinates of corners of the runway(interest points) with unknown runway length and width as follows:P0_(x,z)=(0,0); P1_(x,z)=([P0_(x)+runway_length], P0_(z));P2_(x,z)=([P0_(x)+runway_length], [P0_(z)+runway_width]); andP3_(x,z)=([P0_(x), [P0_(z)+runway_width]). The PnP estimation is alsoperformed as modified to determine the current pose estimate including2DOF made up of the vertical angular deviation OVERT of the aircraftfrom the glideslope 208, and a lateral angular deviation SLAT of theaircraft from the centerline 210 of the runway.

In particular, for example, the runway 202 may be assumed to be arectangular, level plane at unknown runway-framed local coordinates (x,0, z), and an unknown orientation along the z/x axis. The runway mayalso be assumed to have an unknown width. The relation between realworld position and rotation and projected representation, then, may bedescribed with:

$\frac{c - a}{o} = \frac{{{- {\cos(r)}}\left( {{{\cos(q)} \times \left( {- 1} \right)x} - {{\sin(q)}{\cos(p)} \times \left( {- 1} \right)z}} \right)} - {{\sin(r)}{\sin(p)}z}}{{{\sin(q)}x} - {{\cos(q)}{\cos(p)} \times \left( {- 1} \right)z}}$$\mspace{79mu}{\frac{b - d}{o} = \frac{{\sin(r)}\left( {{z{\cos(p)}{\sin(q)}} - {x{\cos(q)}} + {z{\sin(p)}{\cos(r)}}} \right)}{{{- z}{\cos(p)}{\cos(q)}} - {x{\sin(q)}}}}$

In the preceding, (a, b)=pixel position (x, y); (c, d)=screen center (x,y); o=scaling factor; (p, q, r)=rotation angles around (x, y, z);x=reference point world runway-framed local coordinate x; andz=runway-framed local coordinate z. When reducing roll angle to zero(rotation of the input image to level the horizon), it can be shown thatonly two points (e.g. two threshold center points) may be needed for asingle solution for the lateral and (if the runway length is known) thevertical angular deviation. The threshold center points here refer tothe center point of each threshold marking (on per approach end).

Shape-Based Approach. PnP estimation techniques often need two to fourpoints on the runway 202, as well as the runway-framed local coordinates(x, y, z) of those points. The ability to perceive all four corners of arunway may be limited by environmental conditions, runway contamination,obstructions and the like. In accordance with a shape-based approach,the pose-estimation engine 402 may determine the current pose estimate412 without any specific part of the runway visible in the image, andwithout a-priori knowledge of runway parameters. In accordance with theshape-based approach, the mask 410 to be unbounded or not approximatedwith an enclosing polygon. The approach also does not require anyspecific runway points.

The shape-based approach in some examples is premised on scale,translation and orientation invariant moments that describe the shape(mass distribution) of an object and can be made translation invariantor translation and scale invariant. In this regard, invariants η_(ij)with respect to translation and scale may be determined from centralmoments by dividing through a scaled zero-th central moment μ₀₀:

$\eta_{ij} = \frac{\mu_{ij}}{\mu_{00}\left( {1 + \frac{i + j}{2}} \right)}$

In the above, i+j≥2. It may also be noted that translational invariancedirectly follows by only using central moments. For a 2DOF pose estimatemade up of the vertical and lateral angular deviation, the shape-basedapproach may include the pose-estimation engine inferring the currentpose estimate from a runway based on the shape of a binaryrepresentation of a semantically-separated runway.

According to the shape-based approach, in some examples in which theobject detection and segmentation is performed to detect the runway 202,the pose-estimation engine 402 is configured to perform aparameterization of the mask 410 in which a shape of the mask isdescribed by values of at least one of a set of parameters 424 or aninterpolation of the set of parameters. In some examples, the set ofparameters 424 includes moment invariants of the mask 410, and thepose-estimation engine 402 is configured to perform the parameterizationto determine values of the moment invariants of the mask.

In some examples, the pose-estimation engine 402 is configured todetermine the current pose estimate of the aircraft 200 relative to therunway based on the values of the set of parameters and an expression426 that maps the at least one of the set of parameters or theinterpolation of the set of parameters to a pose or an interpolated poseof the aircraft relative to the runway. In various examples, theexpression is implemented as a stochastic algorithm such as a machinelearning algorithm. In other examples, the expression is implemented asa deterministic algorithm. In yet other examples, the expression isimplemented as a lookup table of different values of the set ofparameters and ground truth poses of the aircraft relative to the runway202. In some of these examples, the pose-estimation engine may beconfigured to determine a closest match based on a moment invariantdelta (e.g., closest scale, position or scale, position, orientationinvariant shape match), such as from the lookup table or otherimplementation of the expression.

According to the shape-based approach, then, the shape of the runway 202may be described as a set of parameters such as the moment invariants ofthe mask 410. The pose of the aircraft 200 relative to the runway may bedetermined in a number of different manners based on values of the setof parameters. The pose may be determined by looking up the closest setfrom a library of stored parameters and their associated poses (e.g.,lateral and vertical deviation). The pose may be determined byinterpolating the pose from a library of stored parameters andinterpolating the associated pose. In another example, the pose may bedetermined from a machine learning algorithm trained to map the set ofparameters to the associated pose. And in yet another example, the posemay be determined from a deterministic algorithm such as a mathematicalrelation that maps the set of parameters to the associated pose.

Two-Lines Approach. According to another approach, only two lines thatcorrespond to sides of the runway 202 may be required to determine thecurrent pose estimate 412. The pose-estimation engine 402 may thereforedetermine the current pose estimate without corners of the runwayvisible in the image, and instead use two vectors of unknown magnitudeto determine the current pose estimate.

According to the two-lines approach, in some examples in which theobject detection and segmentation is performed to detect the runway 202,the pose-estimation engine 402 is configured to determine edges of themask 410 that correspond to sides of the runway, and values of spatialdimensions of the edges. FIG. 6A illustrates one example of a mask 600with edges 602 that correspond to sides of a runway.

In some examples, the pose-estimation engine 402 is configured toperform a regression analysis 428 such as a multi-variate linearregression in which a pose of the aircraft relative to the runway isexpressed as a function of parameters that describe spatial dimensionsof the edges. The regression analysis is performed to determine thecurrent pose estimate 412 of the aircraft relative to the runway fromthe values of the spatial dimensions of the edges. The spatialdimensions of the edges may be expressed in a number of differentmanners, such as in slope-intercept form, two-point form or the like.

In some further examples, the pose-estimation engine 402 is configuredto perform the regression analysis in which components of the pose ofthe aircraft 200 are expressed as respective functions of parametersthat describe the spatial dimensions of the edges. In some of theseexamples, the regression analysis is performed to determine values ofthe components and thereby the current pose estimate of the aircraftrelative to the runway 202. In a more particular example implementationin which components of the current pose estimate include altitude (y),pitch and lateral angular offset (SLAT), the components may be expressedas the following respective functions:

y=684.28−1183.2m ₁−1.192b ₁−916.74m ₂−1.064b ₂

pitch=−8.598+16.536m ₁+0.000589b ₁+26.6836m ₂+0.028923b ₂

θ_(LAT)=33.848−35.01m ₁+0.062b ₁−170.081m ₂−0.172b ₂

In the preceding, the spatial dimensions of the edges may be expressedin slope-intercept form (m₁, b₁), (m₂, b₂).

In some examples, the pose-estimation engine 402 configured to performthe regression analysis includes the pose-estimation engine configuredto perform the regression analysis of different statistical models todetermine multiple current pose estimates 412 of the aircraft relativeto the runway 202 from the values of the spatial dimensions of theedges. The pose-estimation engine is configured to determine confidenceintervals associated with respective ones of the multiple current poseestimates, and perform a sensor fusion of the multiple current poseestimates using the confidence intervals to determine the current poseestimate of the aircraft relative to the runway.

Sideline Approach. Yet another approach, the sideline approach, issimilar to the two-lines approach. The sideline approach does notrequire any particular points on the runway or a-priori knowledge ofrunway parameters. According to the sideline approach, in some examplesin which the object detection and segmentation is performed to detectthe runway 202, the pose-estimation engine 402 configured to determinethe current pose estimate includes the pose-estimation engine configuredto determine edges of the mask 410 that correspond to sides of therunway, and angular deviations of the edges from a centerline of theimage. FIG. 6B illustrates the above example of the mask 600 with edges602 that correspond to sides of a runway, and further including angulardeviations θ₁, θ₂ from centerline 604.

In some examples, the pose-estimation engine 402 is configured todetermine the current pose estimate 412 of the aircraft 200 relative tothe runway 202 from the edges and the angular deviations of the edgesfrom the centerline. The current pose estimate here includes two DOFmade up of the vertical angular deviation θ_(VERT) of the aircraft fromthe glideslope 208, and the lateral angular deviation θ_(LAT) of theaircraft from a centerline 210 of the runway. The vertical angulardeviation may be determined by the slope of the edge, a width overdistance (number of pixels) relation, or some combination thereof. Ifthe sum of both edges is positive then the camera 408 is on centerlineand the lateral angular deviation is zero; otherwise, the lateraldeviation may be qualified by the relative sum of the two angles. Inparticular, for example, the vertical angular deviation may be inferredfrom the slope of one of the edges and/or relative distance of the twoedges from each other at predetermined points. The lateral angulardeviation may be inferred from the difference between the angulardeviations of the edges from the centerline.

In some implementations in which a portion of the runway 202 is clipped(not visible) in the image, the mask 410 may include up to six edges, asshown in FIG. 7 for a mask 700. In some of these examples, a shapedetection function may be used to approximate a corresponding polygon,such as by using a recursive, up/down goal-seeking algorithm to controlan accuracy parameter so that an intended number of edges are detected.The pose-estimation engine may be configured to determine the edges ofthe mask that correspond to the sides of the runway as the two longest,not near-horizontal, not near-vertical lines of the mask. In otherimplementations, the pose-estimation engine may be configured todetermine all intersection points of all of the edges of the mask, andonly consider those having a vanishing point near the horizon asprobable sides of the runway.

FIGS. 8A-8F are flowcharts illustrating various steps in a method 800 ofsupporting an aircraft approaching a runway on an airfield, according tosome example implementations. As shown at block 802 of FIG. 8A, themethod includes receiving a sequence of images of the airfield, capturedby at least one camera onboard the aircraft approaching the runway. Themethod includes, for at least one image of the sequence of images,applying the at least one image to a machine learning model trained topredict a pose of the aircraft relative to the runway, as shown at block804. The machine learning model is configured to map the at least oneimage to the pose based on a training set of labeled images withrespective ground truth poses of the aircraft relative to the runway.And as shown at block 806, the pose is output as a current pose estimateof the aircraft relative to the runway for use in at least one ofmonitoring the current pose estimate, generating an alert based on thecurrent pose estimate, or guidance or control of the aircraft on a finalapproach.

In some examples, applying the at least one image to the machinelearning model at block 804 includes applying the at least one image tothe machine learning model trained to predict a pose of the at least onecamera in camera coordinates, as shown at block 808. In some of theseexamples, the camera coordinates are transformed to correspondingrunway-framed local coordinates and thereby predict the pose of theaircraft relative to the runway, as shown at block 810.

In some examples, the at least one image and the labeled images are in anon-visible light spectrum. In other examples, the labeled images aremono-channel images, the at least one image is a multi-channel image,and the method further comprises converting the multi-channel image to amono-channel image that is applied to the machine learning model, asshown at block 812 of FIG. 8B.

In some examples, the method 800 further includes cropping the at leastone image to reduce a field of view of the at least one camera, as shownat block 814 of FIG. 8C. In some of these examples, as shown at block816, the method also includes magnifying only a portion of the at leastone image on which the runway is located, before the at least one imageis applied to the machine learning model at block 804.

In some examples, the method 800 further includes generating thetraining set of labeled images, as shown at 818 of FIG. 8D. In some ofthese examples, generating the training set includes receiving earlierimages of the airfield, captured by the at least one camera onboard theaircraft or a second aircraft approaching the runway, and the respectiveground truth poses of the aircraft or the second aircraft relative tothe runway, as shown at block 820. The earlier images are labeled withthe respective ground truth poses of the aircraft to generate thetraining set of labeled images, as shown at block 822.

In some examples, the method 800 further includes generating thetraining set of labeled images as shown at 824 in FIG. 8E. In some ofthese examples, generating the training set includes executing a flightsimulator configured to artificially re-create flight of the aircraftapproaching the runway on the airfield, as shown at block 826.Generating the training set also includes capturing synthetic images ofthe airfield, and determining the respective ground truth poses of theaircraft relative to the runway, from the flight simulator, as shown atblock 828. And the synthetic images are labeled with the respectiveground truth poses of the aircraft to generate the training set oflabeled images, as shown at block 830.

Briefly returning to FIG. 8A, in some examples, applying the at leastone image to the machine learning model at block 804 includes applyingthe at least one image to machine learning models trained to predictrespective components of the pose of the aircraft relative to therunway. The machine learning models are configured to determine valuesof the components and thereby the pose of the aircraft relative to therunway.

In some examples, applying the at least one image to the machinelearning model at block 804 includes applying the at least one image tomachine learning models trained to predict multiple current poseestimates according to different algorithms. In some of these examples,the method further includes determining confidence intervals associatedwith respective ones of the multiple current pose estimates, as shown atblock 832 of FIG. 8F. Also in some of these examples, the methodincludes performing a sensor fusion of the multiple current poseestimates using the confidence intervals to determine the current poseestimate of the aircraft relative to the runway, as shown at block 834.

FIGS. 9A-9N are flowcharts illustrating various steps in a method 900 ofsupporting an aircraft approaching a runway on an airfield, according toother example implementations. As shown at block 902 of FIG. 9A, themethod includes receiving a sequence of images of the airfield, capturedby at least one camera onboard the aircraft approaching the runway. Themethod includes, for at least an image of the sequence of images,performing an object detection and segmentation in which at least one ofthe runway on the airfield, or a runway marking on the runway, isdetected in the image, and in which a mask is produced that includes asegment of pixels of the image assigned to an object class for therunway or the runway marking, as shown at block 904. A current poseestimate of the aircraft relative to the runway or the runway marking isdetermined based on the mask, as shown at block 906. And the currentpose estimate is output for use in at least one of monitoring thecurrent pose estimate, generating an alert based on the current poseestimate, or guidance or control of the aircraft on a final approach, asshown at block 908.

In some examples, determining the current pose estimate of the aircraftrelative to the runway or the runway marking at block 906 includes atleast determining the current pose estimate of the at least one camerain camera coordinates, as shown at block 910. In some of these examples,the camera coordinates are transformed to corresponding runway-framedlocal coordinates that are output for use in the at least one ofmonitoring the current pose estimate, generating the alert based on thecurrent pose estimate, or guidance or control of the aircraft, as shownat block 912.

In some examples, the sequence of images are in the non-visible lightspectrum. In other examples, as shown at block 914 of FIG. 9B, the imageis a multi-channel image, and the method 900 further includes convertingthe multi-channel image to a mono-channel image on which the objectdetection and segmentation is performed at block 904.

In some examples, the method 900 further includes cropping the image toreduce a field of view of the at least one camera, as shown at block 916of FIG. 9C, In some of these examples, as shown at block 918, the methodalso includes magnifying only a portion of the image on which the runwayor the runway marking is located, before the object detection andsegmentation is performed at block 904.

In some further examples, the method 900 further includes accessing anearlier image in the sequence of images, and an earlier mask includes arespective segment of pixels of the earlier image assigned to the objectclass for the runway or the runway marking, as shown at block 920 ofFIG. 9D. In some of these examples, the method includes identifying aportion of the earlier image that frames the earlier mask, and therebythe portion of the airfield on which the runway or the runway marking islocated, as shown at block 922. And in some of these examples, croppingthe image at block 916 includes cropping the image to a portion of theimage that corresponds to the portion of the earlier image.

In some examples, performing the object detection and segmentation atblock 904 includes applying the image to a machine learning modeltrained to perform the object detection and segmentation, as shown atblock 924 of FIG. 9E.

In some examples, performing the object detection and segmentation atblock 904 includes performing a feature detection in which features ofthe runway or the runway marking are detected in the image, and in whichthe mask is produced from the features, as shown at block 926 of FIG.9F.

Turning now to NG. 9G, in some examples, determining the current poseestimate at block 906 includes at least applying the mask to a cornerdetector to detect interest points on the mask and thereby the runway orthe runway marking in the image, as shown at block 928. The interestpoints on the runway or the runway marking in the image are matched tocorresponding points on the runway or the runway marking that have knownrunway-framed local coordinates, as shown at block 930. And aperspective-n-point (PnP) estimation is performed, using the interestpoints and the known runway-framed local coordinates, to determine thecurrent pose estimate of the at least one camera and thereby theaircraft relative to the runway or the runway marking, as shown at block932.

In some further examples, performing the object detection andsegmentation at block 904 includes applying the image to a machinelearning model trained to perform the object detection and segmentation,as shown at block 934 of FIG. 9H. In some of these examples, the method900 further includes determining an operational state of the aircraft,as shown at block 936 of FIG. 9H. In some of these examples, based onthe operational state, a first machine learning model or a secondmachine learning model is selected as the machine learning model towhich the image is applied, as shown at block 938. The first machinelearning model is trained to perform a one-class object detection andsegmentation, and the second machine learning model is trained toperform a multi-class object detection and segmentation.

In some examples, performing the object detection and segmentation atblock 904 includes applying the image to a machine learning modeltrained to perform the object detection and segmentation, as shown atblock 940 of FIG. 9I. In some of these examples, the machine learningmodel further determines a confidence interval or multi-dimensionalconfidence matrix associated with detection of the runway or the runwaymarking. Also in some of these examples, the method 900 further includesapplying the mask to a confidence filter that passes the mask to thecorner detector only when the confidence interval is above a thresholdconfidence interval that is dynamically set based on an operationalstate of the aircraft, as shown at block 942.

Briefly returning to FIG. 9G, in some examples, the PnP estimation isperformed at block 932 to determine to multiple current pose estimatesof the aircraft relative to the runway or the runway marking. In some ofthese examples, the method 900 further includes determining confidenceintervals associated with respective ones of the multiple current poseestimates, as shown at block 944 of FIG. 9J. Also in some of theseexamples, the method includes performing a sensor fusion of the multiplecurrent pose estimates using the confidence intervals to determine thecurrent pose estimate of the aircraft relative to the runway or therunway marking, as shown at block 946.

In some examples, the object detection and segmentation is performed atblock 904 to detect the runway, and in which the mask includes thesegment of pixels of the image assigned to the object class for therunway. In some of these examples, determining the current pose estimateat block 906 includes at least applying the mask to a corner detector todetect a pair of interest points on the mask and thereby the runway inthe image, as shown at block 948 of FIG. 91K. The pair of interestpoints on the runway in the image is matched to corresponding points onthe runway that have known runway-framed local coordinates, as shown atblock 950.

Also in some of these examples, a PnP estimation is performed for n=2,using the pair of interest points and the known runway-framed localcoordinates, to determine the current pose estimate of the at least onecamera and thereby the aircraft relative to the runway, as shown atblock 952. The PnP estimation is modified based on an assumption thatthe runway is rectangular and planar, and the PnP estimation isperformed as modified to determine the current pose estimate includingtwo degrees-of-freedom (DOF) made up of a vertical angular deviation ofthe aircraft from a glideslope, and a lateral angular deviation of theaircraft from a centerline of the runway.

In some further examples, the object detection and segmentation isperformed at block 904 to detect the runway, and in which the maskincludes the segment of pixels of the image assigned to the object classfor the runway. In some of these examples, determining the current poseestimate at block 906 includes at least performing a parameterization ofthe mask in which a shape of the mask is described by values of at leastone of a set of parameters or an interpolation of the set of parameters,as shown at block 954 of FIG. 9L. Also in some of these examples, acurrent pose estimate of the aircraft relative to the runway isdetermined based on the values of the set of parameters and anexpression that maps the at least one of the set of parameters or theinterpolation of the set of parameters to a pose or an interpolated poseof the aircraft relative to the runway, as shown at block 956.

In some examples, the set of parameters includes moment invariants ofthe mask, and performing the parameterization at block 954 includesdetermining values of the moment invariants of the mask.

In some examples, the current pose estimate is determined at block 956based on the expression that is implemented as a lookup table ofdifferent values of the set of parameters and ground truth poses of theaircraft relative to the runway. In other examples, the current poseestimate is determined based on the expression that is implemented as astochastic algorithm. And in yet other examples, the current poseestimate is determined based on the expression that is implemented as adeterministic algorithm.

In some examples, the object detection and segmentation is performed atblock 904 to detect the runway, and in which the mask includes thesegment of pixels of the image assigned to the object class for therunway. In some of these examples, determining the current pose estimateat block 906 includes at least determining edges of the mask thatcorrespond to sides of the runway, and values of spatial dimensions ofthe edges, as shown at block 958 of FIG. 9M. Also in some of theseexamples, a regression analysis is performed in which a pose of theaircraft relative to the runway is expressed as a function of parametersthat describe spatial dimensions of the edges, as shown at block 960.The regression analysis is performed to determine the current poseestimate of the aircraft relative to the runway from the values of thespatial dimensions of the edges.

In some further examples, performing the regression analysis at block960 includes performing the regression analysis in which components ofthe pose of the aircraft are expressed as respective functions ofparameters that describe the spatial dimensions of the edges. In some ofthese examples, the regression analysis is performed to determine valuesof the components and thereby the current pose estimate of the aircraftrelative to the runway.

In some examples, performing the regression analysis at block 960includes performing the regression analysis of different statisticalmodels to determine multiple current pose estimates of the aircraftrelative to the runway from the values of the spatial dimensions of theedges, as shown at block 962, Also in some of these examples, confidenceintervals associated with respective ones of the multiple current poseestimates are determined, and a sensor fusion of the multiple currentpose estimates is performed using the confidence intervals to determinethe current pose estimate of the aircraft, relative to the runway, asshown in blocks 964 and 966.

In some examples, the object detection and segmentation is performed atblock 904 to detect the runway, and in which the mask includes thesegment of pixels of the image assigned to the object class for therunway. In some of these examples, determining the current pose estimateat block 906 includes at least determining edges of the mask thatcorrespond to sides of the runway, and angular deviations of the edgesfrom a centerline of the image, as shown at block 968 of FIG. 9N. Alsoin some of these examples, the current pose estimate of the aircraft,relative to the runway is determined from the edges and the angulardeviations of the edges from the centerline, at shown at block 970, Thecurrent pose estimate includes two degrees-of-freedom (DOF) made up of avertical angular deviation of the aircraft from a glideslope, and alateral angular deviation of the aircraft from a centerline of therunway.

According to example implementations of the present disclosure, thesystems 300, 400 and their respective subsystems may be implemented byvarious means. Means for implementing the system and its subsystems mayinclude hardware, alone or under direction of one or more computerprograms from a computer-readable storage medium.

In some examples, one or more apparatuses may be configured to functionas or otherwise implement the system and its subsystems shown anddescribed herein. In examples involving more than one apparatus, therespective apparatuses may be connected to or otherwise in communicationwith one another in a number of different manners, such as directly orindirectly via a wired or wireless network or the like.

FIG. 10 illustrates an apparatus 1000 according to some exampleimplementations of the present disclosure. Generally, an apparatus ofexemplary implementations of the present disclosure may comprise,include or be embodied in one or more fixed or portable electronicdevices. Examples of suitable electronic devices include a smartphone,tablet computer, laptop computer, desktop computer, workstationcomputer, server computer or the like. The apparatus may include one ormore of each of a number of components such as, for example, processingcircuitry 1002 (e.g., processor unit) connected to a memory 1004 (e.g.,storage device).

The processing circuitry 1002 may be composed of one or more processorsalone or in combination with one or more memories. The processingcircuitry is generally any piece of computer hardware that is capable ofprocessing information such as, for example, data, computer programsand/or other suitable electronic information. The processing circuitryis composed of a collection of electronic circuits some of which may bepackaged as an integrated circuit or multiple interconnected integratedcircuits (an integrated circuit at times more commonly referred to as a“chip”). The processing circuitry may be configured to execute computerprograms, which may be stored onboard the processing circuitry orotherwise stored in the memory 1004 (of the same or another apparatus).

The processing circuitry 1002 may be a number of processors, amulti-core processor or some other type of processor, depending on theparticular implementation. Further, the processing circuitry may beimplemented using a number of heterogeneous processor systems in which amain processor is present with one or more secondary processors on asingle chip. As another illustrative example, the processing circuitrymay be a symmetric multi-processor system containing multiple processorsof the same type. In yet another example, the processing circuitry maybe embodied as or otherwise include one or more ASICs, FPGAs or thelike. Thus, although the processing circuitry may be capable ofexecuting a computer program to perform one or more functions, theprocessing circuitry of various examples may be capable of performingone or more functions without the aid of a computer program. In eitherinstance, the processing circuitry may be appropriately programmed toperform functions or operations according to example implementations ofthe present disclosure.

The memory 1004 is generally any piece of computer hardware that iscapable of storing information such as, for example, data, computerprograms (e.g., computer-readable program code 1006) and/or othersuitable information either on a temporary basis and/or a permanentbasis. The memory may include volatile and/or non-volatile memory, andmay be fixed or removable. Examples of suitable memory include randomaccess memory (RAM), read-only memory (ROM), a hard drive, a flashmemory, a thumb drive, a removable computer diskette, an optical disk, amagnetic tape or some combination of the above. Optical disks mayinclude compact disk-read only memory (CD-ROM), compact disk-read/write(CD-R/W), DVD or the like. In various instances, the memory may bereferred to as a computer-readable storage medium. The computer-readablestorage medium is a non-transitory device capable of storinginformation, and is distinguishable from computer-readable transmissionmedia such as electronic transitory signals capable of carryinginformation from one location to another. Computer-readable medium asdescribed herein may generally refer to a computer-readable storagemedium or computer-readable transmission medium.

In addition to the memory 1004, the processing circuitry 1002 may alsobe connected to one or more interfaces for displaying, transmittingand/or receiving information. The interfaces may include acommunications interface 1008 (e.g., communications unit) and/or one ormore user interfaces. The communications interface may be configured totransmit and/or receive information, such as to and/or from otherapparatus(es), network(s) or the like. The communications interface maybe configured to transmit and/or receive information by physical (wired)and/or wireless communications links. Examples of suitable communicationinterfaces include a network interface controller (NIC), wireless NIC(WNIC) or the like.

The user interfaces may include a display 1010 and/or one or more userinput interfaces 1012 (e.g., input/output unit). The display may beconfigured to present or otherwise display information to a user,suitable examples of which include a liquid crystal display (LCD),light-emitting diode display (LED), plasma display panel (PDP) or thelike. The user input interfaces may be wired or wireless, and may beconfigured to receive information from a user into the apparatus, suchas for processing, storage and/or display. Suitable examples of userinput interfaces include a microphone, image or video capture device,keyboard or keypad, joystick, touch-sensitive surface (separate from orintegrated into a touchscreen), biometric sensor or the like. The userinterfaces may further include one or more interfaces for communicatingwith peripherals such as printers, scanners or the like.

As indicated above, program code instructions may be stored in memory,and executed by processing circuitry that is thereby programmed, toimplement functions of the systems, subsystems, tools and theirrespective elements described herein. As will be appreciated, anysuitable program code instructions may be loaded onto a computer orother programmable apparatus from a computer-readable storage medium toproduce a particular machine, such that the particular machine becomes ameans for implementing the functions specified herein. These programcode instructions may also be stored in a computer-readable storagemedium that can direct a computer, a processing circuitry or otherprogrammable apparatus to function in a particular manner to therebygenerate a particular machine or particular article of manufacture. Theinstructions stored in the computer-readable storage medium may producean article of manufacture, where the article of manufacture becomes ameans for implementing functions described herein. The program codeinstructions may be retrieved from a computer-readable storage mediumand loaded into a computer, processing circuitry or other programmableapparatus to configure the computer, processing circuitry or otherprogrammable apparatus to execute operations to be performed on or bythe computer, processing circuitry or other programmable apparatus.

Retrieval, loading and execution of the program code instructions may beperformed sequentially such that one instruction is retrieved, loadedand executed at a time. In some example implementations, retrieval,loading and/or execution may be performed in parallel such that multipleinstructions are retrieved, loaded, and/or executed together. Executionof the program code instructions may produce a computer-implementedprocess such that the instructions executed by the computer, processingcircuitry or other programmable apparatus provide operations forimplementing functions described herein.

Execution of instructions by a processing circuitry, or storage ofinstructions in a computer-readable storage medium, supportscombinations of operations for performing the specified functions. Inthis manner, an apparatus 1000 may include a processing circuitry 1002and a computer-readable storage medium or memory 1004 coupled to theprocessing circuitry, where the processing circuitry is configured toexecute computer-readable program code 1006 stored in the memory. Itwill also be understood that one or more functions, and combinations offunctions, may be implemented by special purpose hardware-based computersystems and/or processing circuitry which perform the specifiedfunctions, or combinations of special purpose hardware and program codeinstructions.

As explained above and reiterated below, the present disclosureincludes, without limitation, the following example implementations.

Clause 1. An apparatus for supporting an aircraft approaching a runwayon an airfield, the apparatus comprising: a memory configured to storecomputer-readable program code; and processing circuitry configured toaccess the memory, and execute the computer-readable program code tocause the apparatus to at least: receive a sequence of images of theairfield, captured by at least one camera onboard the aircraftapproaching the runway; and for at least one image of the sequence ofimages, apply the at least one image to a machine learning model trainedto predict a pose of the aircraft relative to the runway, the machinelearning model configured to map the at least one image to the posebased on a training set of labeled images with respective ground truthposes of the aircraft relative to the runway; and output the pose as acurrent pose estimate of the aircraft relative to the runway for use inat least one of monitoring the current pose estimate, generating analert based on the current pose estimate, or guidance or control of theaircraft on a final approach.

Clause 2. The apparatus of clause 1, wherein the apparatus caused toapply the at least one image to the machine learning model includes theapparatus caused to at least: apply the at least one image to themachine learning model trained to predict a pose of the at least onecamera in camera coordinates; and transform the camera coordinates forthe at least one camera to corresponding runway-framed local coordinatesand thereby predict the pose of the aircraft relative to the runway.

Clause 3. The apparatus of clause 1 or clause 2, wherein the at leastone image and the labeled images are in a non-visible light spectrum.

Clause 4. The apparatus of any of clauses 1 to 3, wherein the labeledimages are mono-channel images, the at least one image is amulti-channel image, and the processing circuitry is configured toexecute the computer-readable program code to cause the apparatus tofurther convert the multi-channel image to a mono-channel image that isapplied to the machine learning model.

Clause 5. The apparatus of any of clauses 1 to 4, wherein the processingcircuitry is configured to execute the computer-readable program code tocause the apparatus to further crop the at least one image to reduce afield of view of the at least one camera, and magnify only a portion ofthe at least one image on which the runway is located, before the atleast one image is applied to the machine learning model.

Clause 6, The apparatus of any of clauses 1 to 5, wherein the processingcircuitry is configured to execute the computer-readable program code tocause the apparatus to further generate the training set of labeledimages, including the apparatus caused to at least: receive earlierimages of the airfield, captured by the at least one camera onboard theaircraft or a second aircraft approaching the runway, and the respectiveground truth poses of the aircraft or the second aircraft relative tothe runway; and label the earlier images with the respective groundtruth poses of the aircraft to generate the training set of labeledimages.

Clause 7. The apparatus of any of clauses 1 to 6, wherein the processingcircuitry is configured to execute the computer-readable program code tocause the apparatus to further generate the training set of labeledimages, including the apparatus caused to at least: execute a flightsimulator configured to artificially re-create flight of the aircraftapproaching the runway on the airfield; capture synthetic images of theairfield, and determine the respective ground truth poses of theaircraft relative to the runway, from the flight simulator; and labelthe synthetic images with the respective ground truth poses of theaircraft to generate the training set of labeled images.

Clause 8. The apparatus of any of clauses 1 to 7, wherein the apparatuscaused to apply the at least one image to the machine learning modelincludes the apparatus caused to apply the at least one image to machinelearning models trained to predict respective components of the pose ofthe aircraft relative to the runway, the machine learning modelsconfigured to determine values of the components and thereby the pose ofthe aircraft relative to the runway.

Clause 9, The apparatus of any of clauses 1 to 8, wherein the apparatuscaused to apply the at least one image to the machine learning modelincludes the apparatus caused to apply the at least one image to machinelearning models trained to predict multiple current pose estimatesaccording to different algorithms, and the processing circuitry isconfigured to execute the computer-readable program code to cause theapparatus to further at least: determine confidence intervals associatedwith respective ones of the multiple current pose estimates; and performa sensor fusion of the multiple current pose estimates using theconfidence intervals to determine the current pose estimate of theaircraft relative to the runway.

Clause 10. A method of supporting an aircraft approaching a runway on anairfield, the method comprising: receiving a sequence of images of theairfield, captured by at least one camera onboard the aircraftapproaching the runway; and for at least one image of the sequence ofimages, applying the at least one image to a machine learning modeltrained to predict a pose of the aircraft relative to the runway, themachine learning model configured to map the at least one image to thepose based on a training set of labeled images with respective groundtruth poses of the aircraft relative to the runway; and outputting thepose as a current pose estimate of the aircraft relative to the runwayfor use in at least one of monitoring the current pose estimate,generating an alert based on the current pose estimate, or guidance orcontrol of the aircraft on a final approach.

Clause 11. The method of clause 10, wherein applying the at least oneimage to the machine learning model includes: applying the at least oneimage to the machine learning model trained to predict a pose of the atleast one camera in camera coordinates; and transforming the cameracoordinates for the at least one camera to corresponding runway-framedlocal coordinates and thereby predict the pose of the aircraft relativeto the runway.

Clause 12. The method of clause 10 or clause 11, wherein the at leastone image and the labeled images are in a non-visible light spectrum.

Clause 13. The method of any of clauses 10 to 12, wherein the labeledimages are mono-channel images, the at least one image is amulti-channel image, and the method further comprises converting themulti-channel image to a mono-channel image that is applied to themachine learning model.

Clause 14. The method of any of clauses 10 to 13 further comprisingcropping the at least one image to reduce a field of view of the atleast one camera, and magnifying only a portion of the at least oneimage on which the runway is located, before the at least one image isapplied to the machine learning model.

Clause 15. The method of any of clauses 10 to 14 further comprisinggenerating the training set of labeled images, including at least:receiving earlier images of the airfield, captured by the at least onecamera onboard the aircraft or a second aircraft approaching the runway,and the respective ground truth poses of the aircraft or the secondaircraft relative to the runway; and labeling the earlier images withthe respective ground truth poses of the aircraft to generate thetraining set of labeled images.

Clause 16. The method of any of clauses 10 to 15 further comprisinggenerating the training set of labeled images, including at least:executing a flight simulator configured to artificially re-create flightof the aircraft approaching the runway on the airfield; capturingsynthetic images of the airfield, and determining the respective groundtruth poses of the aircraft relative to the runway, from the flightsimulator; and labeling the synthetic images with the respective groundtruth poses of the aircraft to generate the training set of labeledimages.

Clause 17. The method of any of clauses 10 to 16, wherein applying theat least one image to the machine learning model includes applying theat least one image to machine learning models trained to predictrespective components of the pose of the aircraft relative to therunway, the machine learning models configured to determine values ofthe components and thereby the pose of the aircraft relative to therunway.

Clause 18. The method of any of clauses 10 to 17, wherein applying theat least one image to the machine learning model includes applying theat least one image to machine learning models trained to predictmultiple current pose estimates according to different algorithms, andthe method further comprises: determining confidence intervalsassociated with respective ones of the multiple current pose estimates;and performing a sensor fusion of the multiple current pose estimatesusing the confidence intervals to determine the current pose estimate ofthe aircraft relative to the runway.

Clause 19. An apparatus for supporting an aircraft approaching a runwayon an airfield, the apparatus comprising: a memory configured to storecomputer-readable program code; and processing circuitry configured toaccess the memory, and execute the computer-readable program code tocause the apparatus to at least: receive a sequence of images of theairfield, captured by at least one camera onboard the aircraftapproaching the runway; and for at least an image of the sequence ofimages, perform an object detection and segmentation in which at leastone of the runway on the airfield, or a runway marking on the runway, isdetected in the image, and in which a mask is produced that includes asegment of pixels of the image assigned to an object class for therunway or the runway marking; determine a current pose estimate of theaircraft relative to the runway or the runway marking based on the mask;and output the current pose estimate for use in at least one ofmonitoring the current pose estimate, generating an alert based on thecurrent pose estimate, or guidance or control of the aircraft on a finalapproach.

Clause 20. The apparatus of clause 19, wherein the apparatus caused todetermine the current pose estimate of the aircraft relative to therunway or the runway marking includes the apparatus caused to at least:determine the current pose estimate of the at least one camera in cameracoordinates; and transform the camera coordinates for the at least onecamera to corresponding runway-framed local coordinates that are outputfor use in the at least one of monitoring the current pose estimate,generating the alert based on the current pose estimate, or guidance orcontrol of the aircraft.

Clause 21. The apparatus of clause 19 or clause 20, wherein the sequenceof images are in the non-visible light spectrum.

Clause 22. The apparatus of any of clauses 19 to 21, wherein the imageis a multi-channel image, and the processing circuitry is configured toexecute the computer-readable program code to cause the apparatus tofurther convert the multi-channel image to a mono-channel image on whichthe object detection and segmentation is performed.

Clause 23. The apparatus of any of clauses 19 to 22, wherein theprocessing circuitry is configured to execute the computer-readableprogram code to cause the apparatus to further crop the image to reducea field of view of the at least one camera, and magnify only a portionof the image on which the runway or the runway marking is located,before the object detection and segmentation is performed.

Clause 24. The apparatus of clause 23, wherein the processing circuitryis configured to execute the computer-readable program code to cause theapparatus to further at least: access an earlier image in the sequenceof images, and an earlier mask includes a respective segment of pixelsof the earlier image assigned to the object class for the runway or therunway marking; and identify a portion of the earlier image that framesthe earlier mask, and thereby the portion of the airfield on which therunway or the runway marking is located, and wherein the apparatuscaused to crop the image includes the apparatus caused to crop the imageto a portion of the image that corresponds to the portion of the earlierimage.

Clause 25. The apparatus of any of clauses 19 to 24, wherein theapparatus caused to perform the object detection and segmentationincludes the apparatus caused to apply the image to a machine learningmodel trained to perform the object detection and segmentation.

Clause 26. The apparatus of any of clauses 19 to 25, wherein theapparatus caused to perform the object detection and segmentationincludes the apparatus caused to perform a feature detection in whichfeatures of the runway or the runway marking are detected in the image,and in which the mask is produced from the features.

Clause 27. The apparatus of any of clauses 19 to 26, wherein theapparatus caused to determine the current pose estimate includes theapparatus caused to at least: apply the mask to a corner detector todetect interest points on the mask and thereby the runway or the runwaymarking in the image; match the interest points on the runway or therunway marking in the image, to corresponding points on the runway orthe runway marking that have known runway-framed local coordinates; andperform a perspective-n-point (PnP) estimation, using the interestpoints and the known runway-framed local coordinates, to determine thecurrent pose estimate of the at least one camera and thereby theaircraft relative to the runway or the runway marking.

Clause 28. The apparatus of clause 27, wherein the apparatus caused toperform the object detection and segmentation includes the apparatuscaused to apply the image to a machine learning model trained to performthe object detection and segmentation, and the processing circuitry isconfigured to execute the computer-readable program code to cause theapparatus to further at least: determine an operational state of theaircraft; and based on the operational state, select a first machinelearning model or a second machine learning model as the machinelearning model to which the image is applied, the first machine learningmodel trained to perform a one-class object detection and segmentation,and the second machine learning model trained to perform a multi-classobject detection and segmentation.

Clause 29. The apparatus of clause 27 or clause 28, wherein theapparatus caused to perform the object detection and segmentationincludes the apparatus caused to apply the image to a machine learningmodel trained to perform the object detection and segmentation, and themachine learning model further determines a confidence interval ormulti-dimensional confidence matrix associated with detection of therunway or the runway marking, and wherein the processing circuitry isconfigured to execute the computer-readable program code to cause theapparatus to further apply the mask to a confidence filter that passesthe mask to the corner detector only when the confidence interval isabove a threshold confidence interval that is dynamically set based onan operational state of the aircraft.

Clause 30. The apparatus of any of clauses 27 to 29, wherein the PnPestimation is performed to determine to multiple current pose estimatesof the aircraft relative to the runway or the runway marking, and theprocessing circuitry is configured to execute the computer-readableprogram code to cause the apparatus to further at least: determineconfidence intervals associated with respective ones of the multiplecurrent pose estimates; and perform a sensor fusion of the multiplecurrent pose estimates using the confidence intervals to determine thecurrent pose estimate of the aircraft relative to the runway or therunway marking.

Clause 31. The apparatus of any of clauses 19 to 30, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and the apparatus caused to determinethe current pose estimate includes the apparatus caused to at least:apply the mask to a corner detector to detect a pair of interest pointson the mask and thereby the runway in the image; match the pair ofinterest points on the runway in the image, to corresponding points onthe runway that have known runway-framed local coordinates; and performa perspective-n-point (PnP) estimation for n=2, using the pair ofinterest points and the known runway-framed local coordinates, todetermine the current pose estimate of the at least one camera andthereby the aircraft relative to the runway, the PnP estimation modifiedbased on an assumption that the runway is rectangular and planar, andwherein the PnP estimation is performed as modified to determine thecurrent pose estimate including two degrees-of-freedom (DOE) made up ofa vertical angular deviation of the aircraft from a glideslope, and alateral angular deviation of the aircraft from a centerline of therunway.

Clause 32. The apparatus of any of clauses 19 to 31, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and the apparatus caused to determinethe current pose estimate includes the apparatus caused to at least:perform a parameterization of the mask in which a shape of the mask isdescribed by values of at least one of a set of parameters or aninterpolation of the set of parameters; and determine a current poseestimate of the aircraft relative to the runway, based on the values ofthe set of parameters and an expression that maps the at least one ofthe set of parameters or the interpolation of the set of parameters to apose or an interpolated pose of the aircraft relative to the runway.

Clause 33. The apparatus of clause 32, wherein the set of parametersincludes moment invariants of the mask, and the apparatus caused toperform the parameterization includes the apparatus caused to determinevalues of the moment invariants of the mask.

Clause 34. The apparatus of clause 32 or clause 33, wherein the currentpose estimate is determined based on the expression that is implementedas a lookup table of different values of the set of parameters andground truth poses of the aircraft relative to the runway.

Clause 35. The apparatus of any of clauses 32 to 34, wherein the currentpose estimate is determined based on the expression that is implementedas a stochastic algorithm.

Clause 36. The apparatus of any of clauses 32 to 35, wherein the currentpose estimate is determined based on the expression that is implementedas a deterministic algorithm.

Clause 37. The apparatus of any of clauses 19 to 36, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and the apparatus caused to determinethe current pose estimate includes the apparatus caused to at least:determine edges of the mask that correspond to sides of the runway, andvalues of spatial dimensions of the edges; and perform a regressionanalysis in which a pose of the aircraft relative to the runway isexpressed as a function of parameters that describe spatial dimensionsof the edges, the regression analysis performed to determine the currentpose estimate of the aircraft relative to the runway from the values ofthe spatial dimensions of the edges.

Clause 38. The apparatus of clause 37, wherein the apparatus caused toperform the regression analysis includes the apparatus caused to performthe regression analysis in which components of the pose of the aircraftare expressed as respective functions of parameters that describe thespatial dimensions of the edges, the regression analysis performed todetermine values of the components and thereby the current pose estimateof the aircraft relative to the runway.

Clause 39. The apparatus of clause 37 or clause 38, wherein theapparatus caused to perform the regression analysis includes theapparatus caused to at least: perform the regression analysis ofdifferent statistical models to determine multiple current poseestimates of the aircraft relative to the runway from the values of thespatial dimensions of the edges; determine confidence intervalsassociated with respective ones of the multiple current pose estimates;and perform a sensor fusion of the multiple current pose estimates usingthe confidence intervals to determine the current pose estimate of theaircraft relative to the runway.

Clause 40. The apparatus of any of clauses 19 to 39, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and the apparatus caused to determinethe current pose estimate includes the apparatus caused to at least:determine edges of the mask that correspond to sides of the runway, andangular deviations of the edges from a centerline of the image; anddetermine the current pose estimate of the aircraft relative to therunway from the edges and the angular deviations of the edges from thecenterline, the current pose estimate including two degrees-of-freedom(DOF) made up of a vertical angular deviation of the aircraft from aglideslope, and a lateral angular deviation of the aircraft from acenterline of the runway.

Clause 41. A method of supporting an aircraft approaching a runway on anairfield, the method comprising: receiving a sequence of images of theairfield, captured by at least one camera onboard the aircraftapproaching the runway; and for at least an image of the sequence ofimages, performing an object detection and segmentation in which atleast one of the runway on the airfield, or a runway marking on therunway, is detected in the image, and in which a mask is produced thatincludes a segment of pixels of the image assigned to an object classfor the runway or the runway marking; determining a current poseestimate of the aircraft relative to the runway or the runway markingbased on the mask; and outputting the current pose estimate for use inat least one of monitoring the current pose estimate, generating analert based on the current pose estimate, or guidance or control of theaircraft on a final approach.

Clause 42. The method of clause 41, wherein determining the current poseestimate of the aircraft relative to the runway or the runway markingincludes at least: determining the current pose estimate of the at leastone camera in camera coordinates; and transforming the cameracoordinates for the at least one camera to corresponding runway-framedlocal coordinates that are output for use in the at least one ofmonitoring the current pose estimate, generating the alert based on thecurrent pose estimate, or guidance or control of the aircraft.

Clause 43. The method of clause 41 or clause 42, wherein the sequence ofimages are in the non-visible light spectrum.

Clause 44. The method of any of clauses 41 to 43, wherein the image is amulti-channel image, and the method further comprises converting themulti-channel image to a mono-channel image on which the objectdetection and segmentation is performed.

Clause 45. The method of any of clauses 41 to 44 further comprisingcropping the image to reduce a field of view of the at least one camera,and magnifying only a portion of the image on which the runway or therunway marking is located, before the object detection and segmentationis performed.

Clause 46. The method of clause 45 further comprising: accessing anearlier image in the sequence of images, and an earlier mask includes arespective segment of pixels of the earlier image assigned to the objectclass for the runway or the runway marking; and identifying a portion ofthe earlier image that frames the earlier mask, and thereby the portionof the airfield on which the runway or the runway marking is located,and wherein cropping the image includes cropping the image to a portionof the image that corresponds to the portion of the earlier image.

Clause 47. The method of any of clauses 41 to 46, wherein performing theobject detection and segmentation includes applying the image to amachine learning model trained to perform the object detection andsegmentation.

Clause 48. The method of any of clauses 41 to 47, wherein performing theobject detection and segmentation includes performing a featuredetection in which features of the runway or the runway marking aredetected in the image, and in which the mask is produced from thefeatures.

Clause 49. The method of any of clauses 41 to 48, wherein determiningthe current pose estimate includes at least: applying the mask to acorner detector to detect interest points on the mask and thereby therunway or the runway marking in the image; matching the interest pointson the runway or the runway marking in the image, to correspondingpoints on the runway or the runway marking that have known runway-framedlocal coordinates; and performing a perspective-n-point (PnP)estimation, using the interest points and the known runway-framed localcoordinates, to determine the current pose estimate of the at least onecamera and thereby the aircraft relative to the runway or the runwaymarking.

Clause 50. The method of clause 49, wherein performing the objectdetection and segmentation includes applying the image to a machinelearning model trained to perform the object detection and segmentation,and the method further comprises: determining an operational state ofthe aircraft; and based on the operational state, selecting a firstmachine learning model or a second machine learning model as the machinelearning model to which the image is applied, the first machine learningmodel trained to perform a one-class object detection and segmentation,and the second machine learning model trained to perform a multi-classobject detection and segmentation.

Clause 51. The method of clause 49 or clause 50, wherein performing theobject detection and segmentation includes applying the image to amachine learning model trained to perform the object detection andsegmentation, and the machine learning model further determines aconfidence interval or multi-dimensional confidence matrix associatedwith detection of the runway or the runway marking, and wherein themethod further comprises applying the mask to a confidence filter thatpasses the mask to the corner detector only when the confidence intervalis above a threshold confidence interval that is dynamically set basedon an operational state of the aircraft.

Clause 52. The method of any of clauses 49 to 51, wherein the PnPestimation is performed to determine to multiple current pose estimatesof the aircraft relative to the runway or the runway marking, and themethod further comprises: determining confidence intervals associatedwith respective ones of the multiple current pose estimates; andperforming a sensor fusion of the multiple current pose estimates usingthe confidence intervals to determine the current pose estimate of theaircraft relative to the runway or the runway marking.

Clause 53. The method of any of clauses 41 to 52, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and determining the current poseestimate includes at least: applying the mask to a corner detector todetect a pair of interest points on the mask and thereby the runway inthe image; matching the pair of interest points on the runway in theimage, to corresponding points on the runway that have knownrunway-framed local coordinates; and performing a perspective-n-point(PnP) estimation for n=2, using the pair of interest points and theknown runway-framed local coordinates, to determine the current poseestimate of the at least one camera and thereby the aircraft relative tothe runway, the PnP estimation modified based on an assumption that therunway is rectangular and planar, and wherein the PnP estimation isperformed as modified to determine the current pose estimate includingtwo degrees-of-freedom (DOI) made up of a vertical angular deviation ofthe aircraft from a glideslope, and a lateral angular deviation of theaircraft from a centerline of the runway.

Clause 54. The method of any of clauses 41 to 53, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and determining the current poseestimate includes at least: performing a parameterization of the mask inwhich a shape of the mask is described by values of at least one of aset of parameters or an interpolation of the set of parameters; anddetermining a current pose estimate of the aircraft relative to therunway based on the values of the set of parameters and an expressionthat maps the at least one of the set of parameters or the interpolationof the set of parameters to a pose or an interpolated pose of theaircraft relative to the runway.

Clause 55. The method of clause 54, wherein the set of parametersincludes moment invariants of the mask, and performing theparameterization includes determining values of the moment invariants ofthe mask.

Clause 56. The method of clause 54 or clause 55, wherein the currentpose estimate is determined based on the expression that is implementedas a lookup table of different values of the set of parameters andground truth poses of the aircraft relative to the runway.

Clause 57. The method of any of clauses 54 to 56, wherein the currentpose estimate is determined based on the expression that is implementedas a stochastic algorithm.

Clause 58. The method of any of clauses 54 to 57, wherein the currentpose estimate is determined based on the expression that is implementedas a deterministic algorithm.

Clause 59. The method of any of clauses 41 to 58, wherein the objectdetection and segmentation is performed to detect the runway; and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and determining the current poseestimate includes at least: determining edges of the mask thatcorrespond to sides of the runway, and values of spatial dimensions ofthe edges; and performing a regression analysis in which a pose of theaircraft relative to the runway is expressed as a function of parametersthat describe spatial dimensions of the edges, the regression analysisperformed to determine the current pose estimate of the aircraftrelative to the runway from the values of the spatial dimensions of theedges.

Clause 60. The method of clause 59, wherein performing the regressionanalysis includes performing the regression analysis in which componentsof the pose of the aircraft are expressed as respective functions ofparameters that describe the spatial dimensions of the edges, theregression analysis performed to determine values of the components andthereby the current pose estimate of the aircraft relative to therunway.

Clause 61. The method of clause 59 or clause 60, wherein performing theregression analysis includes: performing the regression analysis ofdifferent statistical models to determine multiple current poseestimates of the aircraft relative to the runway from the values of thespatial dimensions of the edges; determining confidence intervalsassociated with respective ones of the multiple current pose estimates;and performing a sensor fusion of the multiple current pose estimatesusing the confidence intervals to determine the current pose estimate ofthe aircraft relative to the runway.

Clause 62. The method of any of clauses 41 to 61, wherein the objectdetection and segmentation is performed to detect the runway, and inwhich the mask includes the segment of pixels of the image assigned tothe object class for the runway, and determining the current poseestimate includes at least: determining edges of the mask thatcorrespond to sides of the runway, and angular deviations of the edgesfrom a centerline of the image; and determining the current poseestimate of the aircraft relative to the runway from the edges and theangular deviations of the edges from the centerline, the current poseestimate including two degrees-of-freedom (DOF) made up of a verticalangular deviation of the aircraft from a glideslope, and a lateralangular deviation of the aircraft from a centerline of the runway.

Many modifications and other implementations of the disclosure set forthherein will come to mind to one skilled in the art to which thedisclosure pertains having the benefit of the teachings presented in theforegoing description and the associated figures. Therefore, it is to beunderstood that the disclosure is not to be limited to the specificimplementations disclosed and that modifications and otherimplementations are intended to be included within the scope of theappended claims. Moreover, although the foregoing description and theassociated figures describe example implementations in the context ofcertain example combinations of elements and/or functions; it should beappreciated that different combinations of elements and/or functions maybe provided by alternative implementations without departing from thescope of the appended claims. In this regard, for example, differentcombinations of elements and/or functions than those explicitlydescribed above are also contemplated as may be set forth in some of theappended claims. Although specific terms are employed herein, they areused in a generic and descriptive sense only and not for purposes oflimitation.

What is claimed is:
 1. An apparatus for supporting an aircraftapproaching a runway on an airfield, the apparatus comprising: a memoryconfigured to store computer-readable program code; and processingcircuitry configured to access the memory, and execute thecomputer-readable program code to cause the apparatus to at least:receive a sequence of images of the airfield, captured by at least onecamera onboard the aircraft approaching the runway; and for at least oneimage of the sequence of images, apply the at least one image to amachine learning model trained to predict a pose of the aircraftrelative to the runway, the machine learning model configured to map theat least one image to the pose based on a training set of labeled imageswith respective ground truth poses of the aircraft relative to therunway; and output the pose as a current pose estimate of the aircraftrelative to the runway for use in at least one of monitoring the currentpose estimate, generating an alert based on the current pose estimate,or guidance or control of the aircraft on a final approach.
 2. Theapparatus of claim 1, wherein the apparatus caused to apply the at leastone image to the machine learning model includes the apparatus caused toat least: apply the at least one image to the machine learning modeltrained to predict a pose of the at least one camera in cameracoordinates; and transform the camera coordinates for the at least onecamera to corresponding runway-framed local coordinates and therebypredict the pose of the aircraft relative to the runway.
 3. Theapparatus of claim 1, wherein the at least one image and the labeledimages are in a non-visible light spectrum.
 4. The apparatus of claim 1,wherein the labeled images are mono-channel images, the at least oneimage is a multi-channel image, and the processing circuitry isconfigured to execute the computer-readable program code to cause theapparatus to further convert the multi-channel image to a mono-channelimage that is applied to the machine learning model.
 5. The apparatus ofclaim 1, wherein the processing circuitry is configured to execute thecomputer-readable program code to cause the apparatus to further cropthe at least one image to reduce a field of view of the at least onecamera, and magnify only a portion of the at least one image on whichthe runway is located, before the at least one image is applied to themachine learning model.
 6. The apparatus of claim 1, wherein theprocessing circuitry is configured to execute the computer-readableprogram code to cause the apparatus to further generate the training setof labeled images, including the apparatus caused to at least: receiveearlier images of the airfield, captured by the at least one cameraonboard the aircraft or a second aircraft approaching the runway, andthe respective ground truth poses of the aircraft or the second aircraftrelative to the runway; and label the earlier images with the respectiveground truth poses of the aircraft to generate the training set oflabeled images.
 7. The apparatus of claim 1, wherein the processingcircuitry is configured to execute the computer-readable program code tocause the apparatus to further generate the training set of labeledimages, including the apparatus caused to at least: execute a flightsimulator configured to artificially re-create flight of the aircraftapproaching the runway on the airfield; capture synthetic images of theairfield, and determine the respective ground truth poses of theaircraft relative to the runway, from the flight simulator; and labelthe synthetic images with the respective ground truth poses of theaircraft to generate the training set of labeled images.
 8. Theapparatus of claim 1, wherein the apparatus caused to apply the at leastone image to the machine learning model includes the apparatus caused toapply the at least one image to machine learning models trained topredict respective components of the pose of the aircraft relative tothe runway, the machine learning models configured to determine valuesof the components and thereby the pose of the aircraft relative to therunway.
 9. The apparatus of claim 1, wherein the apparatus caused toapply the at least one image to the machine learning model includes theapparatus caused to apply the at least one image to machine learningmodels trained to predict multiple current pose estimates according todifferent algorithms, and the processing circuitry is configured toexecute the computer-readable program code to cause the apparatus tofurther at least: determine confidence intervals associated withrespective ones of the multiple current pose estimates; and perform asensor fusion of the multiple current pose estimates using theconfidence intervals to determine the current pose estimate of theaircraft relative to the runway.
 10. A method of supporting an aircraftapproaching a runway on an airfield, the method comprising: receiving asequence of images of the airfield, captured by at least one cameraonboard the aircraft approaching the runway; and for at least one imageof the sequence of images, applying the at least one image to a machinelearning model trained to predict a pose of the aircraft relative to therunway, the machine learning model configured to map the at least oneimage to the pose based on a training set of labeled images withrespective ground truth poses of the aircraft relative to the runway;and outputting the pose as a current pose estimate of the aircraftrelative to the runway for use in at least one of monitoring the currentpose estimate, generating an alert based on the current pose estimate,or guidance or control of the aircraft on a final approach.
 11. Themethod of claim 10, wherein applying the at least one image to themachine learning model includes: applying the at least one image to themachine learning model trained to predict a pose of the at least onecamera in camera coordinates; and transforming the camera coordinatesfor the at least one camera to corresponding runway-framed localcoordinates and thereby predict the pose of the aircraft relative to therunway.
 12. The method of claim 10, wherein the at least one image andthe labeled images are in a non-visible light spectrum.
 13. The methodof claim 10, wherein the labeled images are mono-channel images, the atleast one image is a multi-channel image, and the method furthercomprises converting the multi-channel image to a mono-channel imagethat is applied to the machine learning model.
 14. The method of claim10 further comprising cropping the at least one image to reduce a fieldof view of the at least one camera, and magnifying only a portion of theat least one image on which the runway is located, before the at leastone image is applied to the machine learning model.
 15. The method ofclaim 10 further comprising generating the training set of labeledimages, including at least: receiving earlier images of the airfield,captured by the at least one camera onboard the aircraft or a secondaircraft approaching the runway, and the respective ground truth posesof the aircraft or the second aircraft relative to the runway; andlabeling the earlier images with the respective ground truth poses ofthe aircraft to generate the training set of labeled images.
 16. Themethod of claim 10 further comprising generating the training set oflabeled images, including at least: executing a flight simulatorconfigured to artificially re-create flight of the aircraft approachingthe runway on the airfield; capturing synthetic images of the airfield,and determining the respective ground truth poses of the aircraftrelative to the runway, from the flight simulator; and labeling thesynthetic images with the respective ground truth poses of the aircraftto generate the training set of labeled images.
 17. The method of claim10, wherein applying the at least one image to the machine learningmodel includes applying the at least one image to machine learningmodels trained to predict respective components of the pose of theaircraft relative to the runway, the machine learning models configuredto determine values of the components and thereby the pose of theaircraft relative to the runway.
 18. The method of claim 10, whereinapplying the at least one image to the machine learning model includesapplying the at least one image to machine learning models trained topredict multiple current pose estimates according to differentalgorithms, and the method further comprises: determining confidenceintervals associated with respective ones of the multiple current poseestimates; and performing a sensor fusion of the multiple current poseestimates using the confidence intervals to determine the current poseestimate of the aircraft relative to the runway.